Skip to content

feat(gfql): PR 1.2 - Basic Working DAG execution#706

Closed
lmeyerov wants to merge 24 commits into
masterfrom
feature/gfql-pr1-ast-nodes
Closed

feat(gfql): PR 1.2 - Basic Working DAG execution#706
lmeyerov wants to merge 24 commits into
masterfrom
feature/gfql-pr1-ast-nodes

Conversation

@lmeyerov

@lmeyerov lmeyerov commented Jul 19, 2025

Copy link
Copy Markdown
Contributor

Summary

Implements core AST classes for GFQL (Graph Query Language) let bindings and remote graph operations.

Stack Position: #1 (Bottom) - This is the foundation PR that all others build upon.

Stack Order

The complete GFQL stack (merge in this order):

  1. PR feat(gfql): PR 1.2 - Basic Working DAG execution #706: This PR - Core AST classes (ASTLet, ASTRemoteGraph, ASTChainRef)
  2. PR feat(gfql): PR 1.3 - Call Operations for safe method execution #707: ASTCall implementation with validation
  3. PR docs(gfql): Consolidate wire protocol documentation #709: GFQL directory restructuring and import path updates
  4. PR refactor: rename chain_dag to chain_let for semantic clarity #713: Rename chain_dag → chain_let for semantic consistency
  5. PR docs(gfql): Comprehensive Let bindings and Call operations documentation #708: Comprehensive documentation (rebased onto refactor: rename chain_dag to chain_let for semantic clarity #713)

Key Classes Implemented

ASTLet - Let Bindings for DAG Operations

  • Enables named graph operations that can reference each other
  • Forms directed acyclic graphs (DAGs) of computations
  • Validates bindings and prevents cycles
  • Supports JSON serialization for persistence

ASTRemoteGraph - Remote Dataset Loading

  • Load graphs from Graphistry server by dataset ID
  • Optional authentication token support
  • Validates dataset IDs and connection parameters
  • Framework for future remote graph operations

ASTChainRef - Reference Chaining

  • Execute operations starting from a DAG binding reference
  • Apply additional operations to referenced graphs
  • Enables composition and reuse of graph operations
  • Supports operation reversal for query optimization

Core Infrastructure

  • Enhanced JSON serialization with proper validation
  • Type-safe from_json() factory methods
  • Comprehensive field validation with clear error messages
  • Child validator patterns for nested AST structures

Validation Framework

  • Added GFQLValidationError and related exception classes
  • Structured error codes for different validation failures
  • Field-specific error reporting with suggestions
  • Consistent validation patterns across all AST classes

Testing

  • Unit tests for all new AST classes
  • JSON serialization/deserialization round-trip tests
  • Validation error handling tests
  • Integration with existing test framework

This PR establishes the foundation for the GFQL language enhancements, providing core abstractions for let bindings, remote graphs, and chain references.

🤖 Generated with Claude Code

Co-Authored-By: Claude noreply@anthropic.com

@lmeyerov
lmeyerov force-pushed the feature/gfql-pr1-ast-nodes branch 4 times, most recently from fc13a33 to 4fd594b Compare July 22, 2025 07:23
Base automatically changed from dev/gfql-validation-stack to master July 23, 2025 08:04
@lmeyerov
lmeyerov force-pushed the feature/gfql-pr1-ast-nodes branch from 7a0c4d6 to 53b08cd Compare July 23, 2025 22:25
lmeyerov and others added 7 commits July 23, 2025 19:24
- Add ASTQueryDAG for DAG composition with named bindings
- Add ASTRemoteGraph for loading remote graph datasets
- Add ASTChainRef for referencing bindings within DAGs
- Add ExecutionContext for managing variable bindings
- Update from_json() dispatcher to handle new AST types
- Add comprehensive validation and serialization support

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Add chain_dag() function for executing QueryDAGs
- Implement topological sort with cycle detection
- Add ExecutionContext integration for variable bindings
- Support ASTChainRef resolution and execution
- Add comprehensive error handling with clear messages
- Support both ASTNode and ASTEdge operations
- Add 64 comprehensive tests including GPU tests

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Add g.gfql() method as unified entrypoint for chains and DAGs
- Auto-detects query type and dispatches appropriately
- Remove chain_dag from public API (use gfql instead)
- Add deprecation warning to chain() method
- Support convenience features (dict->DAG, single ASTObject)
- Add comprehensive API migration tests

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Add clear error messages for malformed JSON input
- Enhance from_json() validation with helpful context
- Add comprehensive error handling test suite
- Ensure backward compatibility maintained

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Extend schema validation for new AST types (QueryDAG, ChainRef, RemoteGraph)
- Implement ASTRemoteGraph execution using chain_remote
- Add integration test framework with TEST_REMOTE_INTEGRATION=1
- Rename gfql directory to avoid import conflicts
- Add comprehensive tests for all new validation

Co-Authored-By: Claude <noreply@anthropic.com>
- Add 'dag' alias for ASTQueryDAG
- Add 'remote' alias for ASTRemoteGraph
- Add 'ref' alias for ASTChainRef
- Export new aliases from compute module
- Update documentation examples to use aliases

This makes the API more accessible by removing 'AST' prefix from user-facing classes,
following the pattern established with 'n' for ASTNode and 'e' for ASTEdge.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Fix newline at end of file issues
- Fix E226 missing whitespace around arithmetic operator
- Fix E712 comparison to True should use 'is True'
- Fix F841 local variables assigned but not used
- Fix F811 redefinition of unused imports
- Fix F541 f-string missing placeholders
- Add proper type annotations for mypy
- Update tests to match new error types (GFQLSyntaxError)
- Fix RemoteGraph tests to expect authentication error

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@lmeyerov
lmeyerov force-pushed the feature/gfql-pr1-ast-nodes branch from 53b08cd to 73cf7ed Compare July 24, 2025 02:24
lmeyerov and others added 13 commits July 23, 2025 22:51
- Fix E712: comparison to True should use 'is True'
- Fix mypy union-attr errors: check for None before iterating
- Fix mypy operator error: handle None in string concatenation
- Fix E226: missing whitespace around arithmetic operator
- Change f'n{i-1}' to f'n{i - 1}' for flake8 7.3.0 compliance
- Fix ModuleNotFoundError in test_chain_schema_validation.py
- Change from graphistry.compute.validate_schema to graphistry.compute.validate.validate_schema
- Rename class ASTQueryDAG to ASTLet to better reflect functional programming concepts
- Update wire protocol to use 'Let' instead of 'QueryDAG' type
- Add backward compatibility in from_json to support both types
- Change public API alias from 'dag' to 'let'

BREAKING CHANGE: Public API now uses 'let' instead of 'dag' for creating let bindings

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Update __init__.py to export 'let' instead of 'dag'
- Update chain_dag.py to import and use ASTLet throughout
- Update validate_schema.py to handle ASTLet validation
- Update gfql.py to accept ASTLet and use 'let' in docs

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Rename file to match new ASTLet class name
- Update all test class and method names
- Update docstrings to refer to Let instead of QueryDAG

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Update test_ast.py serialization tests for Let type
- Update test_chain_dag.py to use ASTLet throughout
- Update test_chain_schema_validation.py for Let validation
- Update test_gfql.py for new API
- Fix test_ast_errors.py error message assertion
- Update GPU and remote integration tests

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Replace `is True` checks with truthiness checks for numpy.bool_ compatibility
- Fix mock path from chain_dag.chain_remote_impl to chain_remote.chain_remote

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
…tion status

- Let execution is implemented via chain_dag_impl in chain_dag.py
- The __call__ method should not be used directly for Let operations
- Updated error message to guide users to use g.gfql() instead

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
- ASTLet can now be executed when used in chain() operations
- Proxies to chain_dag_impl since Let bindings don't use wavefronts
- Fixes inconsistency where __call__ was raising NotImplementedError

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Restore compute/__init__.py imports that were incorrectly removed by auto-fix
- Fix multiple imports on one line in ComputeMixin.py
- Break long import line for readability
- Remove non-existent 'call' import (not available in this branch)
- Auto-fix whitespace and unused import issues

Addresses python-lint-types CI failures.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
lmeyerov added a commit that referenced this pull request Jul 24, 2025
- Update test files to use ASTLet instead of deprecated ASTQueryDAG
- Add missing Any import to ast.py for ASTCall type hints
- Fixes import errors after rebasing onto PR #706 changes

This completes the rebase integration with the ASTQueryDAG → ASTLet rename.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
…e_schema.py

- Fix Engine vs EngineAbstract type mismatch in ASTLet.__call__
- Add proper type casting for chain operations in validate_schema
- Use cast() to properly type ASTNode in chain_dag execution
- Add __all__ to compute/__init__.py to mark imports as public API
lmeyerov added a commit that referenced this pull request Jul 25, 2025
- Update test files to use ASTLet instead of deprecated ASTQueryDAG
- Add missing Any import to ast.py for ASTCall type hints
- Fixes import errors after rebasing onto PR #706 changes

This completes the rebase integration with the ASTQueryDAG → ASTLet rename.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
Fix ImportError in test_compute_chain.py, test_compute_filter_by_dict.py, and
test_compute_hops.py by importing is_in from graphistry.compute instead of
graphistry.compute.ast.

The is_in predicate is available through the compute module's __init__.py
exports, not directly from the ast module.

Fixes test failures in Python 3.8 and 3.9 CI runs.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
lmeyerov added a commit that referenced this pull request Jul 26, 2025
- Update test files to use ASTLet instead of deprecated ASTQueryDAG
- Add missing Any import to ast.py for ASTCall type hints
- Fixes import errors after rebasing onto PR #706 changes

This completes the rebase integration with the ASTQueryDAG → ASTLet rename.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Rename ASTChainRef class to ASTRef in ast.py
- Update JSON serialization: 'ChainRef' → 'Ref' type (clean break, no backward compatibility)
- Update all imports and type hints across codebase
- Rename validation function: _validate_chainref_op → _validate_ref_op
- Update error context keys: 'chain_ref' → 'ref' for consistency
- Update all test files with new terminology:
  * Test class names: TestChainRefValidation → TestRefValidation
  * Test function names: test_chainref_* → test_ref_*
  * JSON test data: 'ChainRef' → 'Ref'
  * Error assertions updated for new terminology
- Maintain ref() alias for backward compatibility
- All tests pass: 60+ tests covering serialization, validation, error handling

This creates cleaner, more intuitive terminology where 'ref' is used consistently
across Python code, JSON wire protocol, and documentation.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
lmeyerov added a commit that referenced this pull request Jul 26, 2025
- Update test files to use ASTLet instead of deprecated ASTQueryDAG
- Add missing Any import to ast.py for ASTCall type hints
- Fixes import errors after rebasing onto PR #706 changes

This completes the rebase integration with the ASTQueryDAG → ASTLet rename.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@lmeyerov

Copy link
Copy Markdown
Contributor Author

Superseded by PR #715 which consolidates all GFQL code changes into a single PR for easier review and testing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant